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ABSTRACT : PURPOSE: To transmit switching timing information to a user without disturbing the 
viewing of main video by deciding the presence/absence of subtitles from the feature 
values of subtitles by the divided areas of image data, performing the reduction/move 
processing of extracted subtitles and compositing them with the main video. 

CONSTITUTION: A main video input part 100 converts an analog video signal inputted 
from a video reproducing device 7 to a digital image signal 5-6. A subordinate video input 
part 200 converts an analog video signal inputted from a television tuner 8 to a digital 
image signal 5-7. A subtitle area extracting part 300 prepares a subtitle area signal 5-20 
by detecting a subtitle area while using a luminance counting means by areas and an 
edge counting means by areas. A video control part 400 extracts the picture elements of 
subtitles from the subordinate video signal 5-7 and the subtitle area signal 5-20, the 
picture elements are thinned and reduced according to the designation of reduction, and 
the position of subtitles is moved according to the designation of place. This processed 
subtitle signal is composited with the main video signal 5-6 and outputted. Further, the 
format of subtitles is detected from the subtitle area signal 5-20 and when it is equal with 
the designation of user, the main video and the subordinate video are switched. 
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Abstract 
Constitution 

A video output device having the following means: a means for extracting the 
characteristic quantity of the subtitle in each divided region from the subtitle video data, a means 
that determines the presence/absence of the subtitle for each region from the characteristic 
quantity of each said divided region, a subtitle pixel extraction means that extracts the subtitle in 
the subtitle region along, two subtitle synthesizing processing means, that is, a subtitle reduction 
means for reducing the subtitle and a subtitle moving means that moves the subtitle, and a video 
synthesizing means that synthesizes the subtitle with the main picture being displayed. Also, for 
switching the main picture and the secondary picture, the video input/output [sic, output] device 
has a format detection means that detects the format of the subtitle from the subtitle region and a 
video switching means that switches the output video. 

Effects 

It is possible to change the size and display position of the synthesized subtitle as desired 
by the user, and it is possible to obtain information indicating whether the main picture and 
secondary picture should be switched without disrupting the viewing of the main picture. 
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Key: 1 Display unit 

7 Video reproduction device 

8 Television tuner 

1 00 Main video input unit 

200 Secondary video input unit 

300 Subtitle region extraction unit 

400 Video control unit 
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Claims 

1. A video output device characterized by the following facts: the video output device has 
plural video signals input from VTR and television tuner and it synthesizes them and output to a 
display unit; this video output device has the following means: a first means for input of the main 
video, a second means for input of the secondary video, a third means for extracting the subtitle 
information from said secondary video assigned by the user with output of the video, and a 
fourth means that processes said secondary video and said main video from the subtitle 
information obtained with said third means and outputs the video signal. 

2. The video output device described in Claim 1 characterized by the fact that said third 
means has a fifth means that determines the presence/absence of said subtitle in each region. 

3. The video output device described in Claim 1 characterized by the fact that said fourth 
means has a sixth means that detects the subtitle pixel of the region which is judged to have said 
subtitle, a seventh means that reduces said detected subtitle, an eighth means that moves said 
subtitle, and a ninth means that synthesizes [said subtitle] to any position of said main picture. 

4. The video output device described in Claim 1 characterized by the fact that said fourth 
means has a tenth means that detects the subtitle format from the region which is judged to have 
said subtitle, and an eleventh means that switches said secondary video to said main video when 
the detected format is in agreement with the format assigned by the user. 

5. The video output device described in Claim 4 characterized by the fact that said tenth 
means detects the format on the basis of whether said subtitle is displayed vertically or 
horizontally and on the basis of the display position in the vertical and horizontal directions. 

6. The video output device described in Claim 2 characterized by the fact that said fifth 
means determines whether said subtitle with conditions including the number of the 
high-luminance pixels and the number of the edges in the picture is present, and whether it is still 
for a specified time. 

Detailed explanation of the invention 
[0001] 

Industrial application field 

The present invention pertains to a video output device that automatically monitors the 
secondary picture of another program so that the user will not miss important scenes of the 
secondary picture while watching the main picture. 
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{0002] 
Prior art 

In recent years, with an increase in the number of the video channels, many video output 
devices have the function of displaying plural pictures on the side of the video receiving system. 

[0003] 

Generally speaking, the conventional multi-video simultaneous output type video output 
devices can be classified into the following three types: In the first system, the secondary picture 
is reduced in the main picture, that is, it is the so-called picture-in-picture system. The second 
system divides the display region (screen) of the video to show the main picture and the 
secondary picture side by side. The third system has text broadcasting instead of the secondary 
picture superimposed on the main picture for a synthesized display. 

[0004] 

For the user of said device, the video channel of the greatest interest is considered the 
main video, while the video channel of lesser interest is considered the secondary video. For 
example, while watching a drama, the user may check a sports program. In this case, the sports 
program is the secondary video so that the user can check the score; the scenes when points were 
scored, scenes featuring popular athletes, etc. 

[0005] 

However, the conventional video output system has various problems associated with the 
watching/listening by the user. 

[0006] 

First of all, in the first system, because the display region of the secondary picture is 
small, it is difficult to check its contents, which is undesirable. Consequently, although the 
secondary picture is output as the same time, important scenes may be missed, so that the desired 
effect cannot be fully realized. 

[0007] 

In the second system, because the display region is bisected, both the main picture and 
the secondary picture are distorted, which is undesirable. Also, when switching from watching 
the main picture to watching the secondary picture, the movement makes it difficult for the user 
to shift his gaze to the new position, which is undesirable. 
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[0008] 

In the third system, text information is used in place of the secondary picture, and the 
user does not view the secondary picture directly. Also, in this system, the? display can be made 
only for programs that have text broadcasting, which is undesirable. 

[0009] 

Problems to be solved by the invention 

The purpose of the present invention is to solve the aforementioned problems of the prior 
art by providing a video output device that allows the user to watching of the main picture 
without missing important scenes of the secondary program, without reducing or distorting the 
main picture, and without diverting of the user's attention. 

[0010] 

Means for solving the problems 

First of all, the method for extracting the subtitle provides a means that extracts the 
characteristic quantity of the subtitle in each divided region from the digital video data, and a 
means for determining the presence/absence of the subtitle for each region from the 
characteristic quantity for each divided region. 

[0011] 

Also, as a method for synthesis of the extracted subtitle to the main video, the present 
invention provides a means for extracting subtitle pixels that extracts only the subtitle in the 
subtitle region, and two subtitle synthesizing processing means, that is, a subtitle reduction 
means that reduces the size of the subtitle and a subtitle moving means that moves the subtitle, 
and a video synthesizing means that synthesizes the subtitle with the main video during viewing. 

[0012] 

Also as a method for switching the main picture and the secondary picture, the present 
invention provides a format detection means that detects the format of the subtitle from the 
subtitle region, and a video switching means that switches the output video. 

[0013] 
Operation 

First of all, the secondary video input with the video input means extracts the 
characteristic quantity in each divided region, and it determines the presence/absence of the 
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subtitle in each region with a subtitle region judging means. As a result, even when the number 
of characters in the subtitle is small for the entire screen, it still allows partial extraction. 

[0014] 

Then, with the subtitle pixel extraction means, only the subtitle with the background 
removed from the region with the subtitle is extracted. The extracted subtitle is processed with 
the subtitle reduction means and subtitle moving means, so that the subtitle can be displayed at 
any position on the screen and with any size. Finally, with the video synthesizing means, the 
main video and the subtitle are synthesized for display on the display unit. Consequently, it is 
possible to acquire the information of the other video without compromising the viewing of the 
main picture, and the user is notified with the timing for switching from the main video to the 
secondary video. 

[0015] 

When the video switching is performed instead of synthesis display, first of all, with the 
format detection means, the format is detected from the extracted subtitle region. Then, with the 
video switching means, it is determined whether the detected format is in agreement with the 
format assigned by the user. If they do agree, the main video and the secondary video are 
switched. As a result, it is possible to switch the video without watching the displayed subtitle, 
so that important scenes are not missed. 

[0016] 

Application examples 

Figure 1 is a block diagram illustrating the overall processing of the present invention. 
Main video input unit (100) inputs the video signal from video reproduction device (7). The 
analog data of the video signal is digitalized and the obtained main video data (5-6) is output. 
Similarly, secondary video input unit (200) inputs the video signal from television tuner (8). The 
analog video signal is digitalized, and secondary video data (5-7) is output. 

[0017] 

Secondary video data (5-7) is input to subtitle region extraction unit (300). Then, 
secondary video data (5-7) is processed, and subtitle region data (5-20) is output. Said subtitle 
region data (5-20) contains data indicating the presence/absence of extracted subtitles in each 
divided region. Also, details of subtitle region extraction unit (300) can be explained with 
reference to Figure 6. 
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[0018] 

Three pieces of data, main video data (5-6), secondary video data (5-7) and subtitle 
region data (20), are input to video control unit (400). It performs synthesis and switching 
processing of videos from said data. Display video data (5-29) output in video control unit (400) 
is output to display unit (1). 

[0019] 

Figure 2 is a block diagram illustrating in detail the processing in video control unit (400). 
The processing in video control unit (400) can roughly be divided into two parts: video synthesis 
processing (400-a) and video switching processing (400-b). Under the instruction of the user, 
control unit (410) switches the processing to video synthesis processing (400-a) or video 
switching processing (400-b). 

[0020] 

First of all, an explanation will be given regarding video synthesis processing (400-a). 
Subtitle region data (5-20) and secondary video data (5-7) are input to subtitle pixel extraction 
unit (420). With subtitle region data (5-20), only those subtitle pixels in the region determined to 
have subtitles with subtitle region data (5-20) are extracted. As a result, subtitle pixel data (5-21) 
excluding the background is output. Also, details of processing of subtitle pixel extraction unit 
(420) can be explained with reference to Figure 15. Subtitle reduction unit (430) and subtitle 
moving unit (440) can prevent subtitle pixel data (5-21) from disrupting the viewing of the main 
picture. Subtitle pixel data (5-21) is input to subtitle reduction unit (430). In this case, the subtitle 
pixels are thinned only when there is an instruction to expand or reduce the subtitle of subtitle 
pixel data (5-21). The data for reducing the subtitle is loaded to change subtitle pixel data (5-21). 
Subtitle pixel data (5-21) is also input to subtitle moving unit (440). The subtitle pixels are 
moved to any location only when the subtitles of subtitle pixel data (5-21) hinders viewing of the 
main picture so that a movement instruction is issued. As a result, the data is loaded to change 
subtitle pixel data (5-21). 

[0021] 

Main video data (5-6) and subtitle pixel data (5-21) are input to video synthesizing unit 
(450). Then, main video data (5-6) and subtitle pixel data (5-21) are synthesized and the 
synthesized data is output as display video data (5-29). As a result, no matter what subtitle is 
displayed on the secondary picture, the subtitle can be added without hindering viewing the main 
picture, and the user can be notified of the time for switching from the main picture to the 
secondary picture. 



8 



[0022] 

In the following, an explanation will be given regarding the processing of video 
switching processing (400-b). First of all, subtitle region data (5-20) is input to format detection 
unit (460). From subtitle region data (5-20), format data (5-22) is detected and output. Also, the 
details of the processing of format detection unit (460) are described with reference to Figures 16 
and 17. Main video data (5-6), format data (5-22) and secondary video data (5-7) are input to 
video switching part (450). It is determined whether format data (5-22) and the format assigned 
by the user are the same. If they are judged to be the same, secondary video data (5-21) is loaded 
in display video data (5-29) for output. As a result, although the user does not watch the subtitles, 
the important scenes still are not missed. 

[0023] 

Figure 3 is a flow chart of the video data in video synthesizing processing (400-a) 
explained with Figure 2. Secondary video data (5-7) are input to subtitle region extraction unit 
(300). Subtitle region extraction unit (300) outputs subtitle region data (5-20) with the region 
with the subtitle extracted. Subtitle region data (5-20) substitutes a 1 in the region containing the 
subtitle, and it substitutes a 0 in the remaining region. Subtitle region data (5-20) and secondary 
video data (5-7) are input to subtitle pixel extraction unit (420). Subtitle pixel extraction unit 
(420) performs processing to extract the subtitle pixels from the region with the subtitle of 
subtitle region data (5-20). The pixel values of secondary video data (5-7) are loaded in the 
subtitle pixels of output subtitle pixel data (5-21). The black value is loaded into the remaining 
pixels. The subtitle pixels are thinned only with subtitle reduction unit (430) only when there is 
an instruction to expand or reduce the subtitle of subtitle pixel data (5-21). In this way, the data 
with the reduced subtitle is loaded to change subtitle pixel data (5-21). Subtitle pixel data (5-21) 
is input to subtitle moving unit (440). The subtitle pixels are moved to any position and the data 
are loaded to change subtitle pixel data (5-21) only when the subtitles of subtitle pixel data 
(5-21) hinder watching of the main picture; thus a move instruction is issued. 

[0024] 

Main video data (5-6) and subtitle pixel data (5-21) are input to video synthesizing unit 
(450). Main video data (5-6) and subtitle pixel data (5-21) are synthesized and output as display 
video data (5-29). The output display video data (5-29) is displayed on display unit (1). As a 
result, regardless of how the secondary picture of the subtitle is displayed, viewing of the main 
picture is not hindered. Also, it is possible to add the subtitle, and to notify the user of the time 
for switching from the main picture to the secondary picture. 
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[0025] 

Figure 4 is a diagram illustrating an example of display after subtitle synthesis with video 
synthesizing processing (400-a) explained with reference to Figure 2. 

[0026] 

In the example of synthesis of (5-29-a), during subtitle synthesis processing (400-a), 
subtitle pixel data (5-21) are not processed. As a result, synthesis display is performed at the 
position of the subtitle displayed in secondary video data (5-7) during input. In (5-29-b), first of 
all, with subtitle reduction unit (420), reduction processing of the subtitle is performed for 
subtitle pixel data (5-21). Also, in this example, subtitle movement processing is performed by 
subtitle moving unit (430) to make a synthetic display. (5-29-c) is an example in which subtitle 
pixel data (5-21) of the region with subtitles is displayed on liquid-crystal panel (1-1). 

[0027] 

Figure 5 is a flow chart of the video data in video switching processing (400-b) explained 
with reference to Figure 2. In this example, secondary video data (5-7) is input to subtitle region 
extraction unit (300). Said subtitle region extraction unit (300) outputs subtitle region data (5-20) 
with the region for which the subtitle is extracted. For subtitle region data (5-20), 1 is loaded in 
the region containing the subtitle, and 0 is loaded in the remaining region. Subtitle region data 
(5-20) is input to format detection unit (460). Format detection unit (460) detects and outputs 
format data (5-22) from the configuration state of the subtitle region. Main video data (5-6), 
format data (5-22) and secondary video data (5-7) are input to video switching part (470). It is 
determined whether format data (5-22) is identical to the format assigned by the user. If they are 
judged to be identical, secondary video data (5-21) is loaded in display video data (5-29) for 
output. Output display video data (5-29) is displayed on display unit (1). 

[0028] 

Figure 6 is a block diagram illustrating the processing performed by subtitle region 
extraction unit (300). As shown in this figure, (5-7) represents the digital video data converted by 
secondary video input unit (200). Characteristic extraction unit (350) extracts the characteristics 
of the subtitle from the input video, and outputs them as the count value. There are two types of 
count values, namely, the value counted by luminance counter (600) for each region and the 
value counted by edge counter (700) for each region. Also, for the processing explained by this 
figure, the image is divided into regions, and each region is processed. 
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[0029] 

Luminance counter (600) for each region extracts the high-luminance pixels in each 
region, and outputs the number. Edge counter (700) for each region extracts the pixels with a 
large color difference from the adjacent pixels (hereinafter to be referred to as significant edge) 
in each region, and outputs the number. The inputs to subtitle region evaluation unit (800) are the 
count values obtained by luminance counter (600) for each region and edge counter (700) for 
each region. When the two count values meet a certain condition, it is determined that there is a 
subtitle in the region. Subtitle region data (5-20) output from subtitle region evaluation unit (800) 
is 1-bit data indicating the presence/absence of subtitles in each region. 1 is loaded in the region 
containing subtitles, while 0 is loaded in the remaining region. 

[0030] 

As subtitle judgment in this application example, both the high luminance pixels and the 
significant pixels as characteristics of the subtitle are checked. Consequently, the high-luminance 
region free of edges, such as in the case of light illumination and the low-luminance region 
having edges, such as checkerboard, are not extracted, so that there is the effect of reducing 
misidentifications. In addition, because the characteristics of the subtitle are judged for each 
divided region, even when the number of characters of the subtitle is small for the overall screen, 
it is still possible to make a partial extraction. 

[0031] 

Figure 7 is a diagram illustrating the program and the data. In this figure, program (5-1) is 
a computer diagram of the extraction synthesis processing of the subtitle shown in Figure 1. Also, 
(5-2) to (5-29) are data used as reference in program (5-1). Of these, (5-2) to (5-5) are parameters 
preset for extraction, and (5-6) to (5-29) represent the data for the work used in one processing 
cycle. 

[0032] 

In the following, an explanation will be given regarding the data. First of all, thresholds 
(5-2), (5-3) are preset thresholds for use in extracting the characteristic quantity of the subtitle 
from the video, and thresholds (5-4), (5-5) are preset thresholds for use in distinguishing the 
subtitle from the characteristic quantity. 

[0033] 

Main video data (5-6) is the digital video data of the newest frame converted by main 
video converter (10), and secondary video data (5-7) is the digital video data of the newest frame 
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converted by secondary video converter (10). For processing, the video data is divided into three 
color components, namely, red video data (5-7-1), green video data (5-7-2), and blue video data 
(5-7-3). 

[0034] 

In the following, an explanation will be given regarding the various types of data 
prepared by characteristics extraction unit (350). Luminance data (5-8) is the configuration data 
with the high-luminance pixels in the video data extracted. Horizontal edge data (5-9) is 
configuration data with the pixels having a large color difference in the horizontal direction of 
the screen extracted, and vertical edge data (5-10) is configuration data with the pixels having a 
large color difference in the vertical direction of the screen extracted. Said frame luminance data 
(5-1 1), preceding frame horizontal edge data (5-12), and preceding frame vertical edge data 
(5-13) are configuration data for temporarily storing luminance data (5-9), horizontal edge data 
(5-10), and vertical edge data (5-1 1) one frame preceding the current frame being processed, 
respectively. 

[0035] 

Luminance checking data (5-14) is configuration data for storing the high-luminance 
pixels together with luminance data (5-8) and preceding frame luminance data (5-1 1). Also, 
horizontal edge checking data (5-15) is configuration data for storing the significant edge pixels 
together with horizontal edge data (5-9) and preceding frame horizontal edge data (5-12). 
Similarly, vertical edge checking data (5-16) is configuration data for storing the pixels of the 
significant edge together with vertical edge data (5-10) and preceding frame vertical edge data 
(5-13). 

[0036] 

Luminance region data (5-17) is configuration data for storing the result of counting the 
number of the high-luminance pixels of luminance checking data (5-14) for each divided region. 
Horizontal edge region data (5-18) is configuration data for storing the number of pixels of 
significant edge of horizontal edge checking data (5-15) for each divided region. Similarly, 
vertical edge region data (5-19) is configuration data for storing the number of pixels of 
significant edge of vertical edge checking data (5-16) for each divided region. Also, it is 
preferred that the individual size of the divided region be that for accommodating one subtitle 
character. In this application example, the screen is divided into 16 sections in the horizontal 
direction and 10 divisions in the vertical direction. 
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[0037] 

In the following, an explanation will be given regarding the data prepared by subtitle 
region evaluation unit (400). Subtitle region data (5-20) is configuration data for storing the 
result of determining the presence/absence of subtitles in each divided region. Also, subtitle pixel 
data (5-21) formed by subtitle pixel extraction unit (900) is video data with only the subtitle 
extracted. 

[0038] 

Format data (5-22) is position and direction data for the subtitle of the extracted 
representative image. Row count data (5-23) is configuration data for storing the number of the 
regions determined to have subtitles. Maximum row count data (5-24) is data for storing the 
number of regions of the row having the maximum number of regions of row count data (5-23). 
Maximum row position data (5-25) is data for storing the row No. of the maximum row. 
Similarly, column count data (5-26) is configuration data for storing the number of regions 
determined to have subtitles in column units. Maximum column count data (5-27) is data for 
storing the maximum value of column count data (5-26). Maximum column position data (5-28) 
is data for storing the column No. of the column having the maximum value. 

[0039] 

Display video data (5-29) is data that uses video control unit (400) to perform video 
synthesis or video switching processing for secondary video data (5-7) for display on the display 
unit. If there are no subtitles in the secondary video data, the main video data is transferred to the 
display video data as is. 

[0040] 

In the following, an explanation will be given in detail regarding the processing of 
luminance counter (600) for each region, edge counter (700) for each region, subtitle region 
evaluation unit (800), and subtitle pixel extraction unit (900) shown in Figure 6, with reference 
to Figure 7. 

[0041] 

Figures 8, 9 and 10 illustrate the processing of luminance counter (600) for each region. 
Figures 11,12, 13 show the processing of edge counter (700) for each region. Figure 14 is a How 
chart illustrating the processing process of subtitle region evaluation unit (800). 
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[0042] 

First of all, the processing process of luminance counter (600) for each region will be 
explained with reference to Figures 8, 9 and 10. Processing steps (601) to (609) (Figure 8) 
extract the high-luminance pixels in the input video. In (601), variables X and Y for use in steps 
(602) to (609) are initialized to 0. In step (602), whether the color components of red video data 
(5-7-1), green video data (5-7-2) and blue video data (5-7-3) of secondary video data (5-7) have 
a luminance with a threshold of 1 or higher is checked. If all of the color components have a 
luminance higher than the threshold of 1, the process goes to step (603). If NOT, it goes to step 
(604). In step (603), 1 is written in the location of luminance data (5-8) at the location equal to 
the location on the image for which the luminance is checked in step (602). In step (604), 0 is 
written in the location of luminance data (5-8) at the location equal to the location on the image 
for which the luminance is checked in step (602). Steps (605) through (609) execute address 
refresh processing for performing said processing for all of the pixels. When all of the processes 
are finished, process control goes to step <6 10). 

[0043] 

Step (610) through step (618) (Figure 9) execute processing for extracting the pixels that 
are continuous in high luminance for plural frames. In step (610), variables X and Y for use in 
steps (611) to (618) are initialized to 0. In step (611), whether both the value of luminance data 
(5-8) and the value of luminance data (5-1 1) of the preceding frame are 1 is checked. If both are 
1, the process goes to step (612). If NOT, it goes to step (613). In step (612), 1 is written to the 
location of luminance checking data (5-14) at the location equal to the location on the luminance 
data checked in step (61 1). In step (613), 0 is written in the location of luminance checking data 
(5-14) at the location equal to the location on the luminance data checked in step (61 1). Steps 
(614) through (618) execute address refresh processing for performing said processing for all of 
the pixels. When all of the processes are finished, process control goes to step (619). 

[0044] 

Steps (619) step (625) (Figure 9) execute processing for refreshing preceding-frame 
luminance data (5-1 1). In step (619), variables X and Y for use in steps (620) to (625) are 
initialized to 0. In step {620), the luminance data is loaded for substitution in the preceding-frame 
luminance data. Steps (621) to (625) execute address refresh processing performed for all of the 
pixels. After the end of all of processing, control goes to step (626). 
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[0045] 

Steps (626) through (639) (Figure 10) execute processing for counting the 
high-luminance pixel number in each divided region. In step (626), variables i, j, Xb, Yb for use 
in steps (627) to (239) [sic; (639)] are initialized to 0. In step (627), whether luminance checking 
data (5-14) is 1 is checked. If the data is 1, control goes to step (628). If NOT, it goes to step 
(629). In step (628), 1 is added to the location of luminance region data (5-17) corresponding to 
the divided regions at the locations of the luminance checking data checked in step (627). In 
steps (629) through (639), address refresh processing is performed for processing all of the pixels. 
After end of all of the processes, the luminance counting result for each region is stored in 
luminance region (5-17). 

[0046] 

In the following, an explanation will be given regarding the processing process of edge 
counter (700) for each region based on Figures 1 1,12,13. Steps (701) through (712) (Figure 1 1) 
execute processing for extracting the pixels of the significant edges in the horizontal and vertical 
directions. In step (701), variables X, Y for use in steps (702) to (712) are initialized to 0. In step 
(702), whether the various color components of red video data (5-7-1), green video data (5-7-2), 
and blue video data (5-7-3) of the input video data have a color difference with a threshold of 2 
or larger in the horizontal direction is checked. If all of the color data have color difference with 
a threshold of 2 or larger, control goes to step (703). If NOT, it goes to step (704). In step (703), 
1 is written in the location of the horizontal edge data at the locations equal to the locations in the 
image for which the color difference has been checked in step (702). In step (704), 0 is written in 
the location of the horizontal edge data at the locations equal to the locations in the image for 
which the color difference has been checked in step (702). Similarly, in steps (705) to (707), the 
vertical edge data is extracted. Steps (708) to (712) execute address refresh processing for 
performing said processing for all of the pixels. After completion of all of the processes, control 
goes to step (713). 

[0047] 

Steps (713) through (324) (Figure 12) execute processing for extracting the pixels that are 
continuous as significant edge for plural frames. In step (713), variables X and Y for use in steps 
(714) to (724) are initialized to 0. In step (714), whether both the value of horizontal edge data 
(5-9) and the value of horizontal edge data (5-12) of the preceding frame are 1 is checked. If both 
are 1, control goes to step (715). If NOT, it goes to step (716). In step (715), 1 is written in the 
location of horizontal edge checking data (5-15) at the location equal to the location on the 
horizontal edge data checked in step (714). In step (716), 0 is written in the location of horizontal 
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edge checking data (5-15) at the location equal to the location on the horizontal edge data 
checked in step (715). Similarly, for the vertical edge checking data, computing is performed in 
steps (717) to (719). Steps (720) through (724) execute address refresh processing for processing 
all of the pixels. When all of the processes are finished, control goes to step (725). 

[0048] 

Steps (725) through (731) (Figure 12) execute processing for refreshing preceding-frame 
horizontal edge data (5-12) and preceding-frame vertical edge data (5-13). In step (725), 
variables X and Y for use in steps (726) to (73 1) are initialized to 0. In step (726), the horizontal 
edge data [5-9] is loaded for substitution in the preceding- frame horizontal edge data (5-15), and 
vertical edge data (5-10) is loaded for substitution in the preceding-frame vertical edge data 
(5-13). Steps (727) to (731) execute address refresh processing for processing all of the pixels. 
After the end of processing, control goes to step (732). 

[0049] 

Steps (732) through (748) (Figure 13) execute processing for counting the pixel numbers 
in the vertical and horizontal significant edges in each divided region. Instep (732), variables i, j, 
Xb, Yb for use in steps (733) to (748) are initialized to 0. In step (733), whether horizontal edge 
checking data (5-15) is 1 is checked. If the data is 1, control goes to step (734). If NOT, it goes to 
step (735). In step (734), 1 is added to the location of horizontal edge region data (5-18) 
corresponding to the divided regions at the locations of the horizontal edge checking data 
checked in step (733). Similarly, the vertical edge region data is computed in steps (735) and 
(736). In steps (737) through (748), address refresh processing is executed for processing all of 
the pixels. After the end of all of the processes, the edge counting results for each region are 
stored in horizontal edge region data (5-18) and vertical edge region data (5-19). 

[0050] 

By means of said processing, the characteristics of the subtitle are extracted for each 

region. 
[0051] 

In the following, an explanation will be given regarding the processing process of subtitle 
region evaluation unit (800) with reference to Figure 14. Steps (801) through step (809) (Figure 
15) execute processing for determining whether there is a subtitle in each divided region. In step 
(801), variables Xb, Yb for use in steps (802) to (809) are initialized to 0. In step (802), whether 
the luminance region data is threshold 3 or larger, the horizontal edge region data is threshold 3 
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or larger, and the vertical edge region data is threshold 3 or larger is checked. If all of the three 
conditions are met, control goes to step (803). If NOT, it goes to step (804). In step (803), 1 is 
written in the location of subtitle region data (5-20) of the regions that are equal to the regions 
checked in step (802). In step (804), a 0 is written in the location of subtitle region data (5-20) of 
the regions equal to the regions checked in step (802). Steps (805) to (809) execute address 
refresh processing for processing all of the pixels. After completion of all of the processes, 
control goes to step (810). As a result of said processing, the presence/absence of subtitle for 
each region is determined. 

[0052] 

In the following, a detailed explanation will be given regarding the processing process of 
subtitle pixel extraction unit (420), format detection unit (460) in Figure 2 with reference to the 
flow chart in Figure 7. Figure 15 is a diagram illustrating the processing process of subtitle pixel 
extraction unit (420). Figures 16 and 17 illustrate processing process of format detection unit 
(460). 

[0053] 

First of all, an explanation will be given regarding subtitle pixel extraction unit (420). 
Steps (420-1) through (420-16) execute extraction of processing the pixels corresponding to the 
subtitles by checking whether each pixel belongs to a given subtitle. In step (420-1), variables i, j, 
Xb, Yb for use in steps (420-2) to (420-16) are initialized to 0. In step (420-2), whether subtitle 
region data (5-20) is 1 is checked. If the data is 1, control goes to step (420-3). If NOT, it goes to 
step (420-05). In step (420-03), whether the red video data, blue video data and green video data 
of the corresponding pixel have a luminance of threshold 1 or higher is checked. If all three are 1 
or more, control goes to step (420-04). If NOT, it goes to step (420-05). In step (420-04), the 
value of the secondary video data is loaded in the location of the pixel checked in step (420-03). 
In step (420-05), 0 substituted in the location of the pixel checked in step (420-03). 

[0054] 

Steps (420-06) to (420-16) execute address refresh processing for processing all of the 
pixels. After completion of all of processes, only the subtitle in the secondary video is extracted. 

[0055] 

In the following, an explanation will be given regarding processing process of format 
detection unit (460). Steps (460-1) through step (460-7) (Figure 16) execute processing for 
projection of the subtitle region data in the row direction. Similarly, steps (460-8) to (460-14) 
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(Figure 16) execute processing of projection of the subtitle region data in the column direction. 
In the following, a detailed explanation will be given regarding processing in the row direction. 
In step (460-1), variables Xb, Yb for use in steps (460-2) to (460-7) are initialized to 0. In step 
(460-2), the numbers of the regions having subtitles is added up for each row. Steps (460-3) to 
(460-7) execute address refresh processing for all of the regions. After completion of all of the 
processes, control goes to step (460-8), and similar processing is performed for projection in the 
column direction. After end of the processing in the column direction, control goes to step 
(460-15). By means of said processing, row count data (5-23) and column count data (5-26) store 
the numbers of the regions having subtitles in the row and column directions, respectively. 

[0056] 

Steps (460-15) through (460-33) (Figure 17) execute processing for combining the results 
of the individual regions and for determining the presence/absence of the final subtitle as well as 
the position and direction of the subtitle. In step (460-15), variables Xb, Yb for use in steps 
(460-16) to (460-25) are initialized to 0. In step (460-16), in order to find the row having the 
largest value among the row count data, whether the row count data is larger than the maximum 
row count data is checked. If larger, control goes to step (460-17). If smaller, it goes to step 
(460-19). In step (460-17), the maximum row count data is replaced with the value of the row 
count data. In step (460-18), the row number at this time is stored as the maximum row position 
data. Steps (460-19) to (460-20) execute address refresh processing for all of the rows. Similarly, 
in steps (46-21) to (460-25), the maximum column count data and the maximum column position 
data are computed. 

[0057] 

In step (460-26), it is determined whether the maximum row count data has a threshold of 
4 or larger or whether the maximum column count data has a threshold of 4 or larger. If YES, it 
is determined that there is a subtitle, and control goes to step (460-27). If NO, it determines that 
there in no subtitle. 

[0058] 

When it is determined that there is a subtitle, it is determined whether the subtitle is 
displayed on the screen in the left column, right column, an upper row or a lower row. 

[0059] 

First of all, in step (460-27), the maximum row count data and the maximum column 
count data are compared. If the maximum row count data is larger, it is determined to be "a row 
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subtitle," and control goes to step (460-28). If the maximum column count data is larger, it is 
determined to be "a column subtitle," and control goes to step (460-31). In step (460-28), it is 
determined whether the maximum row position data is higher than the central row (the 5 row in 
this application example). If YES, it is determined to be "an upper row subtitle," and control 
goes to step (460-29). If NO, it is determined to be "a lower row subtitle," and control goes to 
step (460-30). In step (460-29), the "upper row" is written in the format data. In step (460-30), 
the "lower row" is written in the format data. On the other hand, in step (460-31), it is 
determined whether the maximum column position data is larger than the central column (the 8 
column in this application example). If YES, it is determined to be "right column," and control 
goes to step (460-32). If NO, it is determined to be "left column," and control goes to step 
(460-33). In step (460-32), "right column" is written in the format data. In step (460-33), "left 
column" is written in the format data. 



[0060] 

During the aforementioned processing, it is possible to evaluate the format of the subtitle. 
In this application example, the format is determined by forming divisions horizontally and 
vertically. As the number of divisions is increased, it is possible to make a more detailed 
evaluation of the format. One may also adopt the following scheme: the number of subtitle 
regions is also counted. When the number of regions is large, the proportion of the subtitle on the 
screen is large. When the region number is small, the proportion of the subtitle on the screen is 
small. In this way, the format can be evaluated. 



[0061] 

The subtitle characteristics extraction unit and the subtitle judgment part was explained 
above in detail with reference to flow charts. 

[0062] 

In this subtitle extraction example, with the characteristics extraction unit, it is 
determined whether the subtitle appears in the divided region from the digital video data. 
Consequently, even when the number of characters of the subtitle is small for the overall screen, 
it is still possible to realize a partial extraction. Also, with the subtitle region evaluation unit, 
judgment is made on the information indicating the presence/absence of subtitles projected in the 
row and column directions. Consequently, it is possible to make distinguish whether the 
appearing subtitle is in column or row format. In addition, the characteristics extraction unit 
counts the number of pixels with high luminance over the first threshold contained in each region 
and the number of the edges above the second threshold contained in each region. With this 
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characteristic, the subtitle is not only evaluated when a high-luminance image appears. Also, 
since the edges are determined in several directions, it is possible to extract the complexity of the 
text portion with high Fidelity. 

[0063] 

Effects of the invention 

According to the present invention, since the subtitle extracted from the secondary video 
is synthesized with the main video for viewing, it is possible to obtain the text information as a 
substitute for the secondary video even when text broadcasting is not performed. Also, it is 
possible to change the size and display position of the synthesized subtitle as desired by the user, 
without hindering viewing of the main picture. 

[0064] 

In addition, as the secondary video is switched for display and viewing depending on the 
format of the subtitle, the important scenes will not be missed even when the user does not look 
at the subtitle. 

Brief description of the figures 

Figure 1 is a block diagram illustrating the overall processing of the present invention. 

Figure 2 is a processing block diagram illustrating in detail video control unit (400). 

Figure 3 is a flow chart of the video data in video synthesis processing (400-a) explained 
in Figure 2. 

Figure 4 is a diagram illustrating an example of display synthesized with the subtitle in 
video synthesis processing (400-a) explained with reference to Figure 2. 

Figure 5 is a flow chart of the video data in video switching processing (400-b) explained 
with reference to Figure 2. 

Figure 6 is a block diagram illustrating the processing of subtitle region extraction unit 

(300). 

Figure 7 is a diagram illustrating the program and the data. 

Figure 8 is a flow chart illustrating the processing process of luminance counter (600) in 
each region. 

Figure 9 is a flow chart illustrating the processing process of luminance counter (600) in 
each region. 

Figure 10 is a flow chart illustrating the processing process of luminance counter (600) in 
each region. 
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Figure 1 1 is a flow chart illustrating the processing process of edge counter (700) in each 

region. 

Figure 12 is a flow chart illustrating the processing process of edge counter (700) in each 

region. 

Figure 13 is a flow chart illustrating the processing process of edge counter (700) in each 

region. 

Figure 14 is a flow chart illustrating the processing process of subtitle region evaluation 
unit (800). 

Figure 15 is a flow chart illustrating the processing process of subtitle pixel extraction 
unit (420). 

Figure 16 is a flow chart illustrating the processing process of format detection unit (460). 
Figure 17 is a flow chart illustrating the processing process of format detection unit (460). 

Explanation of the part numbers 

1 Display unit 

1 - 1 Liquid-crystal panel 

7 Main video reproduction device 

8 Television tuner 

100 Main video input unit 

200 Secondary video input unit 

300 Subtitle region extraction unit 

400 Video controller 
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Figure 1 



Key: 1 Display unit 

7 Main video reproduction device 

8 Television tuner 

100 Main video input unit 

200 Secondary video input unit 

300 Subtitle region extraction unit 

400 Video controller 
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Key: 410 Control unit 



420 Subtitle pixel extraction unit 

430 Subtitle reduction unit 

440 Subtitle moving unit 

450 Video synthesis unit 

460 Format detection unit 

470 Video switching unit 
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Key: a 1 st table 

300 Subtitle region extraction unit 

420 Subtitle pixel extraction unit 

430 Subtitle reduction unit 

440 Subtitle moving unit 

450 Video synthesis unit 
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Subtitle region extraction unit 
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Key: 350 Characteristics extraction unit 

600 Luminance counter for each region 
700 Edge counter for each region 
800 Subtitle region evaluation unit 
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Red video data [Y][X] > threshold 1, and 
green video data [Y][X] > threshold 1, and 
blue video data [Y][X] > threshold 1? 
Luminance data [Y][X] = 1 
Luminance data [Y][X] = 0 
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Key: 611 Luminance data [Y][X] = 1, and 

preceding-frame luminance data [Y][X] = 1? 

612 Luminance checking data [Y][X] = 1 

613 Luminance checking data [Y][X] = 0 

620 Preceding-frame luminance data [Y][X] = luminance data [Y][X] 



28 



f 

| i-O.J-P.Xb-O.Yt-0 \ 



630 



Kb » X»»j} «e-Z= { X»< 16 ) 



63) 




Key: 627 
628 



Figure 10 

Luminance checking data |W24+j)lXb»20+i| = i? 
Luminance region data |Yb||Xb] = i 
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|Red video data [Y][X+1] - red video data [Y][X-1]| > threshold 2, and 
|green video data [Y][X+1] - green video data [Y][X-1]| > threshold 2, and 
jblue video data [Y][X+1] - blue video data [Y][X-1]| > threshold 2? 
Horizontal edge data [Y][X] = 1 
Vertical edge data [Y][X] = 0 

|Red video data [Y+1][X] - red video data [Y-1][X]| > threshold 2, and 
|green video data [Y+1][X] - green video data [Y-1][X]| > threshold 2, and 
|blue video data [Y+1][X] - blue video data [Y-1][X]| > threshold 2? 
Horizontal edge data [Y][X] = 1 
Vertical edge data [Y][X] = 0 
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Figure 12 

Key: 7 1 4 Horizontal edge data [ Y] [X] = 1 , and 

preceding-frame horizontal edge data [Y][X] =1? 

715 Horizontal edge checking data [Y][X] = 1 

7 1 6 Horizontal edge checking data [Y] [X] = 0 

717 Vertical edge checking data [Y][X] = 1 
7 1 9 Vertical edge checking data [ Y] [X] = 0 

726 Preceding-frame horizontal edge data [Y][X] = horizontal edge data [Y][X] 
Preceding-frame vertical edge data [Y][X] = vertical edge data [Y][X] 
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Key: 733 Horizontal edge checking data tW2«4j]|Xb"2<Wl - i? 

734 Horizontal edge region data [Yb][Xb]+ = 1 

735 Vertical edge checking data |Yb*24*jUXb*20 + i] = 1? 

736 Vertical edge region data [Yb][Xb] + = 1 



32 



| Xb»0. Yb^Q | 



802 




803 



S04 



l xu«o I 

809 808 

w , x *~ 

^Yb»Yb^tM ^ < YbclQ ) 

Figure 14 



Key: 802 Luminance region data [Yb][Xb] > threshold 3, and 

horizontal edge region data [Yb][Xb] > threshold 3, and 
vertical edge region data [Yb][Xb] > threshold 3? 

803 Subtitle region data [Yb][Xb] = 1 

804 Subtitle region data [Yb][Xb] = 0 
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Key: 420-2 Subtitle region data [Yb][Xb] = 1? 

420-3 Red video data rVb*24+j]tXb*20+i| > threshold 1, and 

Blue video data I Yb'24+jUXb*2<W) > threshold 1 , and 

Green video data lYb*24+jtfXb*20*i| > threshold 1? 
420-4 Subtitle pixel data lYf24+jl[Xb*20+M = secondary video data I Yb*24+j||Xb'2<Hi| 
420-5 Subtitle pixel data lYb»24+j)(Xb'20*i) = 0 
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Row count data [Yb] = row count data [Yb] + subtitle region data [Yb][Xb] 
Column count data [Xb] = column count data [Xb] + subtitle region data 
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Key: 460-16 Roe count data [Yb] > maximum row count data? 
460-17 Maximum row count data = row count data [Yb] 
460-18 Maximum row position data = Yb 

460-21 Column count data [Xb] > maximum column count data? 
460-22 Maximum column count data = column count data [Xb] 
460-23 Maximum column position data = Xb 

460-26 Maximum row count data > threshold 4, or maximum column count data > 
threshold 4? 

460-27 Maximum row count data > maximum count data? 

460-28 Maximum row position data > 5? 

460-29 Subtitled data = "upper row" 

460-30 Subtitled data = "lower row" 

460-3 1 Maximum column position data > 8? 

460-32 Subtitled data = "right column" 

460-33 Subtitled data = "left column" 

A Subtitle present 
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